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Abstract 

In  this  paper,  a  procedure  based  on  the  delete-1  cross-validation  is  given  for  estimating 
the  number  of  super  imposed  exponential  signals,  its  limiting  behavior  is  explored  and 
it  is  shown  that  the  probability  of  overestimating  the  true  number  of  signals  is  larger 
than  a  positive  constant  for  sample  size  large  enough.  Also  a  general  procedure  based 
on  the  cross-validation  is  presented  when  the  deletion  precedes  according  to  a  collection 
of  subsets  of  indices.  The  result  is  similar  to  the  delete-1  cross-validation  if  the  number 
of  deletion  is  fixed.  The  simulation  results  are  provided  for  the  performance  of  the 
procedure  when  the  collections  of  subsets  of  indices  are  chosen  as  those  suggested  by  [7] 
in  a  linear  model  selection  problem. 


^Research  partially  supported  by  the  Natural  Sciences  and  Engineering  Research  Council  of  Canada. 


1.  Introduction 


Detection  of  signals  transmitted  in  the  presence  of  noise  arises  in  communications, 
radio  location  of  objects,  seismic  signal  processing  and  computer-assisted  medical  diag¬ 
nosis,  etc.  Many  procedures  for  determining  the  number  of  signals  can  be  found  in  the 
literature.  Generally  speaking,  except  those  based  on  the  Bayesian  approach,  they  may 
be  classified  into  three  types.  The  procedures  of  the  first  type  are  based  on  the  informa¬ 
tion  theory  which  have  two  components:  one  measures  the  goodness  of  fit  and  another 
penalizes  the  overestimation,  such  as  AIC  (see  [2]).  The  procedures  of  the  second  type 
aim  to  decide  the  number  of  signals  by  minimizing  the  description  length,  which  was 
explained  in  [5,  6].  The  procedures  of  the  third  type  are  based  on  the  idea  of  minimizing 
the  prediction  error;  the  use  of  the  cross-validation  approach  is  among  them.  There  is 
no  definite  classification  of  these  procedures.  Some  are  proposed  based  on  information 
theory,  but  they  may  also  be  counted  as  or  are  asymptotically  equivalent  to  procedures 
of  the  second  type,  and  vice  versa,  e.g.  MDL.  The  same  comments  apply  to  the  relation¬ 
ships  between  the  procedures  of  the  second  and  third  types  or  between  the  procedures 
of  the  first  and  third  types.  Among  all  procedures,  those  based  on  the  cross-validation 
approaches  are  intuitively  attractive.  In  this  paper,  we  shall  discuss  the  performance 

of  a  procedure  which  decides  the  number  of  super  imposed  exponential  signals  by  the 
cross-validation  approach. 

The  linear  model  selection  problem  and  the  problem  of  determination  of  the  num¬ 
ber  of  signals  are  closely  related.  Recently  there  appeared  some  papers  on  the  cross- 
validation  approach  in  a  linear  model  selection  problem.  The  main  reason  for  the  activity 
now  is  that  the  heavy  computation  required  by  the  cross-validation  approach  is  no  more 
an  obstacle  in  today’s  modern  computing  era.  At  this  point  it  is  worthy  to  describe 
the  contributions  made  by  Prof.  Shao  on  this  subject.  In  [7],  the  performance  of  the 
cross-validation  approach  in  a  linear  model  selection  problem  was  discussed,  where  it 
was  shown  that  the  procedure  was  not  consistent  if  a  fixed  number  of  observations  was 
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deleted  for  the  purpose  of  the  cross-validation  and  it  was  discovered  that  instead  of 
deleting  a  fixed  number  of  observations,  if  the  number  of  deletion  tended  to  infinity 
at  certain  rate,  the  resulting  procedure  would  be  consistent.  The  result  was  justified 
theoretically  and  its  good  small  sample  performance  was  supported  by  simulation.  [8] 
further  discussed  the  case  that  the  number  of  regressors  is  not  finite.  The  linear  model 
selection  problem  is  simpler  than  the  nonlinear  one  we  consider  here.  It  is  important 
to  explore  the  performance  of  the  cross-validation  approach  in  the  determination  of  the 

number  of  super  imposed  exponential  signals  with  a  number  of  deletion  fixed  or  varied 
with  the  sample  size. 

Consider  the  following  undamped  exponential  signal  model 

PO 

y{n)  =  ^aje^i'^^-w{n),  n  =  l,...,N,  (1.1) 

where  t  =  v^,  is  a  set  of  unknown  complex  amplitudes,  {uj}  is  a  set  of  unknown 
angular  frequencies,  {u;(n)}  is  a  sequence  of  independently  and  identically  distributed 
complex  random  noise  variables,  and  po  is  unknown.  Hereon,  it  is  assumed  that  po  is 
bounded  by  some  known  finite  number  P.  With  this  model  there  are  two  problems,  the 
determination  of  po  and  the  estimation  of  the  parameters  a/s  and  w/s.  It  was  suggested 
in  [4]  to  use  the  cross  validation  to  estimate  the  po  when  the  sample  size  is  relatively  small. 
To  obtain  the  cross-validation  error  for  a  particular  number  of  signals  p,  it  is  required  to 
estimate  efficiently  a^’s  and  w/s  in  presence  of  the  missing  observations.  Such  a  method 
can  be  found  in  [3]  where  an  efficient  procedure  for  estimating  the  non-linear  parameters, 
namely  w/s,  is  provided  when  one  or  more  observations  are  missing.  Note  that  once  an 
efficient  estimates  of  the  non-linear  parameters  are  found,  the  estimation  of  the  linear 

parameters  can  be  obtained  by  simple  linear  regression.  [1]  is  another  reference  on  this 
subject. 

In  this  paper,  the  details  of  the  procedure  based  on  the  delete- 1  cross-validation  is 
given  in  Section  2,  a  general  procedure  based  on  the  cross-validation  with  the  number 
of  deletion  fixed  or  varied  with  the  sample  size  will  be  presented  in  Section  3,  and  the 
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simulation  results  are  provided  in  Section  4  along  with  the  discussion. 


2.  The  delete- 1  cross-validation 
Throughout  the  paper,  it  is  assumed  that 

^(u;(l))  =  0,  E{w{l)^))=a‘^^  £'|u;(l)|'‘ <  oo,  (2.1) 

is  unknown,  and  the  true  number  of  super  imposed  exponential  signals  is  po- 


Denote  for  p  <  P, 


Y  = 


y{p  + 1)  y{p) 

yip  +  ‘2')  vip  +  i) 


2/(1) 

2/(2) 


[l/u  2/2)  •  •  •  5  Vn-pYi 


I  y{N)  y{N-l)  ...  y{N-p) 

and  _  iy{l), . . .  ,y{N)y.  Let  A(^_n,N,p)  be  the  matrix  obtained  from  the  ma¬ 
trix  Y  with  the  rows  having  the  observation  y{n)  removed,  =  {n'  :  {y{n'  + 

P)j  »2/(^))  is  a  row  of  and  r(^^n,N,p)  be  the  number  of  rows  in  the  matrix 

A-n,N,p)-  For  convenience,  we  will  omit  N  in  the  above  notations. 

Write  =  (7m/’^^)>  and  let  >  . . .  >  be  the 

eigenvalues  of  f  and  6'-"”  =  (SJ-"  -). . . . ,  S^e.p)).  be  a  unit  eigenvector  of  f 

corresponding  to  Denote  the  solutions  of  =  0  by 


j 


j  =  1, . . .  ,p,  where  p^j  >  0.  Define 


^(-n,p) 


e 


Let  Air(_„_p)  be  a  matrix  with  the  nth  row  removed  from  X^-np),  and 

2/(-n.v)  =  (y(l),  •  •  • ,  2/(^  -  1),  y(n  +  1), . . . ,  y{N)y. 

Denote  the  least  square  estimates  of  a  =  (o,. . . . ,  c^y  using  samples  and  y,„, 

by 

<^(-n,p)  —  (.^(_„^p)J^(_,i_p))  ^^(*_„,p)y(^). 
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(2.2) 


respectively  and  write  X(^n,p)  = 


oinujp 

5  .  .  .  ,  O 

N 


).  Let 


CR(p,N)  =  (1/N)  X;  ll!/(>!)  - 


n,p)\\  5 


n=l 


for  p  =  If  CR{p,N)  =  mini<p<p C R{p, N) ,  the  procedure  of  the  delete-1 

cross-validation  approach  estimate  of  po  is  given  by  p. 

The  investigation  on  the  limiting  behavior  of  the  procedure  for  p  >  po  is  carried  on 
as  follows: 

Denote  =  cr^Ip+i  +  0,DD*Q*  =  (7^]),  where  D  =  diag(Q:i, . . . ,  ap^)  and 

1  •••  1 


n 


otWl 


gjpi^i  . . .  g»PWpo 

Write  fp  =  (iV  —  p)~^Y*Y.  By  Lemma  3.2  of  [1],  it  follows  that 


e*‘*'po 


(2.3) 


fp  =  r(P)  +  O(0oglog(iV  -  p)/{N  -  p)),  a.s. 


(2.4) 

(2.5) 


f(-n,p)  =  +0(^loglogr(_„,p)/r(_„,p)),  a.s. 

Following  the  proof  of  Lemma  3.2  of  [1],  it  can  be  shown  that 

Tp  -  f(-n,p)  =  0{1/N).  (2.6) 

Let  >  •  •  •  >  X^ph  be  the  eigenvalues  of  F^^),  A^^^  >  •  •  •  >  A^i  be  the  eigenvalues  of 
fp,  and  >  •  •  •  >  b®  i^be  eigenvalues  of  f  (-7i,p)-  By  [9], 

p+1 

'£{xf  -  Xfy  <  trace(fp  -  F^p))^,  (2.7) 


J=1 


P+1 


-  Af  <  trace(f(_„,,  -  rW)^ 


i=i 


(2.8) 


Let  =  (b^\ . . . ,  be  an  unit  eigenvector  of  fp  corresponding  to  Ap+j.  By  (2.4), 
(2.5),  (2.7)  and  (2.8),  with  appropriate  choices  of  and  b^\  we  have  that  forp  >  po, 


(6' 0')',  b^^ 


{b'  0')', 


4 


where  b  is  an  unit  eigenvector  of  corresponding  to  its  smallest  eigenvalue  and  0 
is  a  vector  of  p  —  po  zeros. 

Denote  the  solutions  of  Yfj=o  =  0  by  j  =  1, . . .  ,p,  where  >  0, 

€  [0, 2n),  j  =  1, . . .  ,p.  When  p  >  po  and  with  an  appropriate  ordering,  it  is  easy  to 
see  that 


1, 


''(P) 

Pj 


1,  U 


(-n,p) 


UJ. 


]■> 


vs  (p) 

uY 

3 


u 


(2.9) 


for  J  =  1, . . .  ,po  and 


(2.10) 


for  A:  =  Po  +  1, . . . ,  p. 


Let9(_„,p)  —  ^®(n,p)  andQ:(_n,p)  —  (X*_„,p)-^(-n,p))  ^■^(-n,p)2/(iv) 

for  n  =  1, . . . ,  iV.  From  the  fact  that  (X(*_„,p)-^(-n,p))~^  =  ^(1/-^))  it  is  easy  to  see  that 


lim  max9(_„,p)  =  0,  for  any  p. 

N-^oo  n<N  ^  ' 


Since  the  square  error  is 


||y(n)  X(n,p)tt(_„,p)|| 

=  11(1,  -i(„, p)(;t(_„,p).f(-,,,))-‘X(-„, )(!/("),!/(.„, A,))'lp 

~  11(1  ~  ?(-n,p))  ^[?/(^)  “■  ®(n,p)^(-n,p)]ll^> 


it  follows  that 

CR{p,N)  =  {1/N)  ^2  11(1  “  9(-n,p))  ^  [2/(^)  ”  ®(n.p)^(-n,p)]  II  • 

n=l 

As  (1  —  g(-n,p))"^  =  1  +  25(_„,p)  +  0(g(„„^j), 

CR(J,,N)  =  (1/N)  •£  ll<i(-n,p)l|"  +  (1/lV)  E[29(-«,p)  +0(^-n,))lll<^(-PJ.)ll^ 

n—1  n=l 

where  d(^-n,p)  =  y{n)  —  x*n,p)^{-n,p)-  By  (2.6)  and  the  fact  that  the  roots  of  a  polynomial 
are  continuous  functions  of  its  coefficients,  it  follows  that 

|X(-„,p)(V,-_„,rtA(_„,p))->X('_„,rt  -  l(p,(A,-„;e(p))-'Ai;„l  =  o(i/JV). 
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Therefore,  for  p>  po, 


CR{p,  N)  -  CR(p„  N)  = 

-\-{l / N'jw ^p  N)D{p)'^ (p,N)  ~  {po,N)^(po)'^ (potN^)  ^pi^/^)i 

where  th(p,;v)  =  [/  -  PxJViN)  with  Pa  =  A{A*A)-^A\  D^p)  =  diag(?(i,p), . . .  ,g(;v.p)), 
9(n,p)  =  ^(„,p)(A’(*p)X(p))-^^(,,_p),  and  for  n  = 

Hence,  by  (2.9)  and  (2.10), 

Pr{CR{p,  N)  —  CR{po,  N)  <0)  >  c  +  o(l),  for  a  constant  c  >  0, 

which  means  that  the  procedure  does  not  select  the  true  number  of  super  imposed 
exponential  signals  with  probability  one. 

3.  The  delete-/;:  cross-validation 


Let  r  be  an  index  set  and  A(^^t,n,p)  be  the  matrix  obtained  from  the  matrix  Y 
with  the  rows  having  the  observations  y(n),  n  ^  t  removed,  9(^-r,N,p)  =  {n  '•  {y{n  + 
p), . . . ,  y{n))  is  a  row  of  A^_r,N,p)},  and  r(^^r,N,p)  be  the  number  of  rows  of  A{-r,N,p)-  For 
convenience,  we  omit  N  in  the  above  notations. 


Denote  f(_.r,p)  =  r(“4,p)^(_r,p)^(-T,p)  =  and  let  >  •  •  •  >  be  the 

eigenvalues  of  f'(-T,p)  and  i>'  =  (So~^’^\ . . . ,  6^“^’^^)'  be  a  unit  eigenvector  of  f  (-^.^p) 

corresponding  to  Denote  the  solutions  of  =  0  by 

j  =  l,...,p,  where  >  0.  Write 


’■*P) 


Xr- 


i-'r,P) 


^.p) 


g*U/p 


X(_T,p)  is  a  matrix  with  the  nth  row  removed  from  .^(-T,p)  for  all  n  €  r,  and  y(^-r,N)  is  a 
random  vector  with  the  nth  observation  removed  from  y{N)  for  all  n  G  r.  Define 


^{-r,p)  —  (■^(-t,p)-^(-t,p))  -X’(_.r,p)y(-T,iV)) 


(3.1) 
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and  for  t  =  {j\  <  ...  <  ji],  write 


.  .  .s  (— r,p) 

. . .  ebi4  ' 

II 

•  •  -  (“‘’■jP) 

... 

and  y{r,N)  is  a  random  vector  with  the  nth  observation  removed  from  y{N)  for  all  n 
Suppose  that  Bis  a,  collection  of  \B\  subsets  of  {1, ... ,  N}  that  have  size  k.  Hereon,  [Ql 
denotes  the  number  of  elements  in  the  set  Q.  Let 

for  p  =  0, . . . ,  P.  If  CR{p,  k,  B,  N)  =  mini<p<p  CR{p,  k,  B,  N),  the  procedure  based  on 
the  delete- A:  cross-validation  approach  uses  p  to  estimate  po- 

B  can  be  chosen  as  the  collection  of  all  possible  subsets  of  N}  that  have  size 

k.  If  this  is  the  case,  then  the  procedure  above  is  the  standard  delete-A:  cross-validation 
and  it  can  be  observed  that  the  probability  of  overestimating  po  is  greater  than  c  >  0  for 
large  N  by  the  same  argument  as  in  Section  2.  [7]  suggested  to  select  B  with  \B\  =  0{N) 
and  the  following  conditions  satisfied:  (a)  every  n,  1  <  n  <  iV,  appears  in  the  same 
number  of  subsets  in  B\  and  (b)  every  pair  {n,m),  1  <  n  <  m  <  N,  appears  in  the  same 
number  of  subsets  in  B.  Since  the  “balanced”  collection  B  is  hard  to  obtain,  [7]  provided 
a  simple  and  easy  method  by  Monte  Carlo,  it  proceeds  as  follows:  randomly  draw  (with 
or  without  replacement)  a  collection  Q  of  \Q\  subsets  of  {!,..., iV}  that  have  size  k 
with  1^1“^ (AT  —  k)~‘^N‘^  =  o(l)  and  determine  the  number  of  super  imposed  exponential 
signals  by  minimizing 

CR{p,  k,  Q,  N)  =  ^2  l|y(r,iv)  ~  -^(t,p)^(-t,p)1I^- 

Their  performances  are  studied  by  Monte  Carlo  simulation  in  the  next  section. 

4.  Simulation 
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The  simulations  for  procedures  in  previous  sections  are  carried  out  for  the  following  two 
super  imposed  signal  model: 

y{n)  =  Q!i  exp(m27r/i)  +  0:2  exp(m27r/2)  +  to(n),  n  =  1, . . . , 

where  /i  =  1/4,  /2  =  1/24  and  ai  =  l/\/2  +  i/>/2,  0:2  =  1-  The  signal  noise  ratio 
(SNR)  is  defined  as 

1  In'  12 

5A^i^  =  l01og,o-E^-• 

In  the  simulation  the  number  of  possible  signals  P  is  given  as  5  and  the  SNR  is  42.  For 
a  given  index  set  r  with  k  elements  the  matrix  may  be  constructed  in  the  worst 

case  scenario  when 

N  -  p  >  {k +  l){p+l).  (4.1) 

This  places  a  limitation  on  the  rate  of  increase  of  k  with  that  of  N.  In  the  all  subset 
case  (the  number  of  ’’/c-out”  subsets  is  ^Ck)  the  inequality  must  be  satisfied.  For  the 
Monte  Carlo  method  this  inequality  may  be  overlooked  at  times  due  to  the  randomness 
of  the  selection  of  the  k  elements. 

There  are  two  types  of  simulations,  one  with  the  incomplete  block  design  and  the 
other  using  the  Monte  Carlo  method  (without  replacement).  The  blocks  were  found 
by  using  the  finite  geometry  constructed  in  the  form  of  N  =  +  q  +  1  where  ^  is  a 

prime.  This  gives  q  +  1  elements  in  a  block.  Each  pair  of  elements  appear  in  only 
one  block  and  each  element  appears  in  exactly  q  +  1  blocks.  In  this  case  one  has  the 
q  +  1  deletions  of  elements  and  \B\  =  N.  Clearly  with  this  kind  of  block  design  the 
inequality  (4.1)  is  satisfied.  For  the  Monte  Carlo  procedure,  N,  k  and  Q  are  chosen 
so  that  |^|“^(7V  —  k)~'^N‘^  decreases  as  N  increases.  For  each  size  N  and  subset  of  k 
deletions  the  number  of  repetitions  is  200  in  both  simulations. 

Table  1 


Sample  Size 

57 

133 

183 

307 

381 

553 

871 

993 

1407 

No.  of  Deletions 

8 

12 

14 

18 

20 

24 

30 

32 

38 

Freq.  of  Right  Detection 

.12 

.10 

.14 

.14 

.13 

.11 

.07 

.09 

.10 
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The  block  design  method  of  specifying  the  subset  for  deletions  appears  to  be  per¬ 
forming  poorly  in  the  simulations.  Its  frequency  of  correct  detection  is  given  in  Table  1 
along  with  the  size  N  and  number  of  deletion  k.  On  the  other  hand  the  Monte  Carlo 
procedure  seems  to  give  better  results  if  the  number  of  deletions  is  chosen  to  be  large. 
The  frequency  of  right  detection  for  this  procedure  is  plotted  in  Figure  1  where  the  ratio 
of  k/N  is  also  shown.  The  curves  clearly  show  that  when  the  ratio  increases  so  does 
the  frequency.  This  simulation  result  suggests  that  to  obtain  consistency  of  estimates, 
the  ratio  o{  k/N  needs  to  increase  with  N  and  this  would  agree  with  the  linear  model 
selection  problem  considered  in  [7]. 


5.  Conclusion 

The  intuitively  appealing  cross-validation  technique  for  estimating  the  number  of 
exponential  signals  is  considered.  For  fixed  number  of  k  deletions  {k  >  1),  it  is  proved 
that  there  is  a  positive  probability  to  overestimate.  This  result  agrees  with  that  of  the 
linear  regression  model  studied  by  [7].  In  [7],  it  is  shown  that  in  order  to  obtain  a 
consistent  linear  model  selection  criterion  one  needs  to  vary  the  deletions  in  such  a  way 
that  the  number  of  deletions  should  increase  with  the  sample  size.  Contrary  to  the  linear 
model,  there  is  a  restriction  on  the  way  that  the  deletions  are  chosen  here.  To  get  good 
estimate  of  the  number  of  signals,  the  number  of  deletions  should  be  large  and  yet  the 
matrix  A^-r,p)  may  be  constructed.  This  tendency  was  supported  by  the  Monte-Carlo 
simulations. 
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